1. Field of the Invention
This invention relates in general to embedded media markers, which signify the existence of media associated with a part of a document containing the embedded media marker, and specifically, to the use of embedded media barcode links.
2. Description of the Related Art
Although paper is one of the most widely used devices for viewing information, it cannot play dynamic media such as video and audio. On the other hand, cell phones are increasingly used to play audio and video but cannot match paper's high resolution, large display size, flexibility in spatial organization, outdoor-readability and robustness for static content. It is now possible to combine the two, using image recognition technology to link paper documents to corresponding dynamic media. A cell phone camera is used to capture an image of a document patch. The document patch is identified using features in the image, and digital media linked to that location in the document is retrieved and then played on the cell phone.
A common method for creating this type of media link on a paper document is to print markers on the document. One obvious example is bar code (Rekimoto, J. and Ayatsuka, Y., CyberCode: designing augmented reality environments with visual tags. In Proceedings of ACM DARE 2000, pp. 1-10.) However, existing barcode printing requires an exclusive space and thus may interfere with the document content layout. One solution that alleviates this issue is by merging data cells with the user-specified image background, which still requires an opaque black-white border for the decoder to locate the data cells. Data Glyphs (Hecht D. L., Embedded Data Glyph Technology for Hardcopy Digital Documents. SPIE-Color Hard Copy and Graphics Arts III, Vol. 2171, pp. 341-352.) overcome these problems by printing a nearly invisible machine-recognizable pattern on the paper. However, this type of marker requires high resolution printers and cameras to identify document locations. Electronic markers like RFID can be used too (Reilly, D., M. Rodgers, R. Argue, et al., Marked-up maps: combining paper maps and electronic information resources. Personal and Ubiquitous Computing, 2006. 10(4): pp. 215-226.), but this approach increases the production costs.
Other systems compute features of the document content itself for identifying the document patch and thus creating a media link. HotPaper (Erol, B., Emilio Antunez, and J. J. Hull. HOTPAPER: multimedia interaction with paper using mobile phones. In Proceedings of ACM Multimedia'08, pp. 399-408.) and Mobile Retriever (Liu, X. and D. Doermann, Mobile Retriever: access to digital documents from their physical source. Int. J. Doc. Anal. Recognit., 2008. 11(1): pp. 19-27) use features based on document text such as the spatial layout of words. Other systems such Bookmarkr (Henze, N. and S. Boll. Snap and share your photobooks. In Proceedings of ACM Multimedia'08, pp. 409-418) and MapSnapper (Hare, J., P. Lewis, L. Gordon, and G. Hart. MapSnapper: Engineering an Efficient Algorithm for Matching Images of Maps from Mobile Phones. Proceedings of Multimedia Content Access: Algorithms and Systems II, 2008) use pixel level image features, such as the SIFT algorithm (Lowe, D. G., Distinctive Image Features from Scale-Invariant Keypoints. Int. J. Comput. Vision, 2004. 60(2): pp. 91-110.), to recognize generic document content such as pictures and graphic elements. With these systems, exclusive spaces are not required for marker printing.
Both marker-based methods and document-appearance-based methods fall short in providing visual guidance for users. Although bar codes and Data Glyphs are visible, they do not directly indicate the existence or type of media associated with them. When appearance-based feature are used, there is no on-paper indication at all to the user that there is media linked to the document. As a result, a HotPaper user has to pan a camera phone over the paper document to look for hotspots until feedback such as a red dot or vibration is presented on the cell phone.
To solve this problem, researchers augment paper with meaningful awareness-mark, called EMM (Liu, Q., Liao, C., Wilcox, L., Dunnigan, A., and Liew, B. 2010. Embedded media markers: marks on paper that signify associated media. In Proceeding of the 14th international Conference on intelligent User interfaces (Hong Kong, China, Feb. 7-10, 2010). IUI '10. ACM, New York, N.Y., 149-158), which indicate the existence, type, and capture guidance of media links. On seeing an EMM, the user knows to capture an image of the EMM-signified document patch with a cell phone in order to view associated digital media. This is analogous to Web pages that use underlines, font differences, or image tags to indicate the existence of links that users then click for additional information. Unlike barcodes, EMMs are nearly transparent and thus do not interfere with the document appearance. Unlike Embedded Data Glyphs or Anoto patterns found in Digital Paper, EMMs can be printed with a regular low-resolution printer and identified from an image captured by a normal cell phone camera. Unlike other appearance-based approaches, EMMs clearly indicate signified document patches and locations. The design of EMMs also indicates what type of media (e.g. audio, video, or image) is associated with the EMM-signified document location. Furthermore, by requiring the captured image to cover the whole mark, the EMM system can improve feature construction accuracy, matching accuracy, and efficient resource usage.
EMM solves many past problems. However, the EMM patch identification is slower than traditional barcode. It also requires the underlying document has enough features for patch identification. Additionally, the scalability and error correction ability test for EMM is still not available. Moreover, some barcode users may doubt the clean document patch has enough information to link to associated media.